
{/* banner prompt: A playful, polished 3D clay-render in matte sage green, cream and terracotta: a stylized NVIDIA RTX 3060 graphics card, and a cute rounded clay whale with a faint glowing cyan wireframe pattern on its side, swimming above a row of three small rounded screens each showing a tiny version of the same whale, tiny clapperboard and soft concentric sound-wave props, soft studio shadows, simple pale cream background, gentle depth of field, warm muted palette, no text, no people, 16:9. */}

Our homepage has an electric whale — a glowing cyan wireframe humpback swimming through a digital ocean — playing behind the headline. It was made for a wide desktop screen, so on a phone it gets cropped to a sliver. We wanted a proper **portrait** version for mobile, and it turned into a small experiment worth writing down: take *one* keyframe and hand it to three different video models — one running locally on our RTX 3060, and two frontier models in the cloud — then judge them honestly.

The twist, as usual, is that "cheapest," "best-looking," and "the one we actually shipped" were three different answers.

## The setup: one keyframe, three animators

Everything starts from a single still. We generated a 9:16 keyframe on the same RTX 3060 that runs the rest of this series — a cyan low-poly wireframe whale over a glowing data grid, with a magenta flare and purple depth to match the original. Then that identical image went, unchanged, into each of the three models as the first frame. Same whale, same starting composition — only the motion model changes.

That is the fair way to compare image-to-video: hold the input constant and watch what each model *does* with it.

## MiniMax H3 — local, free, and it actually works

First, the home team. [MiniMax H3](/blog/2026/08/04-now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/) is an open-weights model we already run locally, streaming its 50-odd GB of weights through a 12GB card a slice at a time.

<YouTube id="xaG_naHmBPg" />

The result is genuinely good: the whale swims with a real, gentle tail undulation and the model even grows its own drifting data-particles by the end. It is coherent, on-style, and it cost nothing but electricity and patience — **about 20 minutes for a 3-second clip at 512×896**. That is the honest local trade-off: free and completely private, but slow, shorter, and lower-resolution than the cloud. As a proof that a consumer card can do this at all, it's a quiet triumph.

## Veo 3.1 — the calm, cinematic one

Next, Google's Veo 3.1 Fast, through the Gemini API, from the exact same keyframe.

<YouTube id="k-lnccxtRPM" />

This is where the jump in production value shows. In **under a minute** and for **a dollar or two**, Veo returned a smooth **720×1280, 8-second** clip — and, unprompted, built an entire digital ocean around the whale: a rippling grid, parallax depth, drifting stars. The motion is calm and ambient, which matters when the video lives *behind* a headline. It never fights the text. This is the one we shipped.

## Sora 2 — the most beautiful, and the one on a clock

Finally, OpenAI's Sora 2, same keyframe again.

<YouTube id="00272-n56_Q" />

On pure craft, Sora 2 makes the best whale of the three. The wireframe mesh is the most refined, and the motion is the most lifelike — a graceful full-body dive with fins and flukes that move like they have weight. It was also the cheapest of the cloud runs (~$0.80 for 8 seconds) and quick (~2 minutes).

There are two catches. First, that gorgeous, dramatic motion is *busier* — stunning as a hero, harder to sit quietly behind text. Second, and decisively for us: **OpenAI's Sora 2 video API stops accepting requests on 24 September 2026.** Building a banner on an API that disappears next week is not a plan.

## A quick colour lesson

One thing the experiment taught us: image models have opinions. Our first keyframes came out pure cyan on black — the model kept stripping the magenta flare and purple wash that give the original its identity. No amount of prompting fixed it; the model simply preferred cyan. The fix was to stop arguing and **composite the magenta back in** — a flare and a purple gradient added to the keyframe in a few lines of code. Every model then faithfully carried that palette through the animation. Sometimes the right move is to give the model a picture, not an argument.

## The verdict

| | Where | Best at | The catch |
|---|---|---|---|
| **MiniMax H3** | Local, RTX 3060 | Free, private, no API | Slow, short, lower-res |
| **Veo 3.1** | Google, cloud | Calm, cinematic, ambient | Paid per second |
| **Sora 2** | OpenAI, cloud | Most beautiful, most lifelike | API shuts down 24 Sep 2026 |

If you want free and private, the local model already does it. If you want the most beautiful single clip, Sora 2 wins — for just a few more days. But for a banner that has to loop quietly behind a headline *and* still be regenerable next month, **Veo 3.1 was the right tool** — and it's the whale now swimming on our homepage when you open [tyolab.com](https://www.tyolab.com) on your phone.

The best-looking model and the right one are rarely the same tool. We keep relearning that, one whale at a time.


79fe4348d5fa7fd1a4e55fd884ded88badb8423f1b073f29c27be5c89a948967

A hands-on comparison of MiniMax H3 (local, RTX 3060), Google Veo 3.1 and OpenAI Sora 2 for image-to-video — from one shared keyframe — judged on motion, fidelity, speed, cost and which API will still exist next week.

We wanted a portrait version of the electric whale on our homepage, so we animated the same AI-generated keyframe with a local model on our RTX 3060 and two frontier cloud models. Free-and-private, best-and-sustainable, and best-looking turned out to be three different answers.

One Whale, Three Video Models: MiniMax H3 vs Veo 3.1 vs Sora 2

{/* banner prompt: A playful, polished 3D clay-render illustration in a soft matte palette of sage green, warm cream and terracotta orange. The scene shows a chunky, rounded, stylized NVIDIA graphics card sculpted from smooth modelling clay, sitting on a flat cream surface. From one end of the card a long ribbon of pale paper unfurls and coils gently into a neat spiral stack, like a scroll of memory. A tiny terracotta pot holding a small green bonsai tree with rounded clay leaves rests on top of the card. Everything has a tactile hand-sculpted matte finish with gently rounded edges, soft diffuse studio lighting, warm subtle shadows, shallow depth of field, and a simple uncluttered pastel background. Wide 16:9 composition, centred, clean and minimal. */}

Last month I put a [27-billion-parameter reasoning model on this same RTX 3060](/blog/2026/08/18-now-it-thinks-a-27b-language-model-on-the-same-rtx-3060/) — Qwen3.8-27B, at 4-bit, about 18GB. It worked, but only just: a bit more than half the model sat on the card and the rest streamed from system RAM, and it typed back at roughly reading speed, about five words a second.

This post is the same model — and a much better story. Someone re-compressed Qwen3.8-27B down to **under 6GB**, small enough that it now fits *entirely* on the 12GB card with room to spare. It runs about **four times faster**. And with the space freed up, it can hold something the 4-bit version never could: a genuinely enormous context — up to **262,000 tokens**, a fat novel's worth of text, read all at once.

## The machine (still the same one)

Same box as every post in this series: an RTX 3060 with **12GB of VRAM**, 64GB of system RAM, Debian. Nothing upgraded. The card is the constant; only what we ask of it changes.

## What changed: ternary instead of 4-bit

Last time, quantization squeezed each of the model's numbers down to 4 bits. This model goes further — it uses **ternary** quantization, where almost every weight is stored as just one of three values: -1, 0, or +1. That is a hair under two bits per weight, and it drops the file from over 50GB in full precision to **5.95GB**.

The payoff is that the model no longer has to be split between the card and RAM. All of it lives on the GPU, where the math is fast, and none of it has to be streamed across from system memory. That single change — everything on-card instead of 57% on-card — is most of where the speed-up comes from.

## The honest numbers

Same model, same card, last month versus this month:

<Table data={
    {
        headers: ['', 'Last month (4-bit)', 'This month (ternary)'],
        rows: [
            ['Size on disk', '~18 GB', '5.95 GB'],
            ['Runs on GPU', '~57% on card, rest from RAM', '100% on card'],
            ['VRAM used', '11 of 12 GB', '~8 GB (at a 32K context)'],
            ['Speed', '~4.7 words/sec', '~20 words/sec (about 27 tokens/sec)'],
            ['Math, coding, reasoning', 'Correct', 'Correct'],
        ]
    }
} />

Twenty words a second is a comfortable reading pace that no longer feels like waiting. For a single thoughtful answer on your own machine, it is genuinely pleasant to use.

## The part you actually asked about: how much can it remember?

"Context" is how much text a model can hold in its head at once — the question you ask, plus every document, transcript, or code file you paste in with it. The bigger the context, the more it can read before it starts forgetting the start.

Once the weights only take ~6GB, the rest of the 12GB card is free for exactly this. So I measured it: how large a context actually fits before the card runs out of memory. The lever is the *KV cache* — the running memory the model keeps of everything it has read so far — and you can store that cache at different precisions, just like the weights.

<Table data={
    {
        headers: ['KV cache precision', 'Largest context that fits', 'VRAM used', 'Speed'],
        rows: [
            ['16-bit (default)', '~64K tokens (about 48,000 words)', '~10.3 GB', '~27 tokens/sec'],
            ['8-bit', '128K tokens (about 96,000 words)', '~11.1 GB', '~26 tokens/sec'],
            ['4-bit', '262K tokens (about 196,000 words)', '~12 GB (tight)', '~26 tokens/sec'],
        ]
    }
} />

So the answer is: on a 12GB card with the GPU to itself, you comfortably get a **64K-token** context out of the box, **128K** by storing the cache at 8-bit, and the model's full **262K** by dropping the cache to 4-bit — right up against the edge of the card's memory. That top figure is roughly 196,000 words: a long novel, a year of meeting notes, or a thick stack of contracts, all in one prompt.

Two things worth noting. First, the speed barely moves as the context grows — the model reads a huge prompt about as fast as a small one, because most of its layers use a lighter form of attention that does not balloon with length. Second, the 4-bit cache is the trade-off you pay for the very largest window: less precise memory of what it read, and only a few hundred megabytes of headroom left. For everyday use, the 8-bit / 128K setting is the sweet spot.

## Does squeezing it that hard break it?

That was my worry — a model stored at under two bits per weight sounds like it should be a lobotomised version of itself. It is not. On the checks I ran it stayed sharp: a multi-step arithmetic word problem came out correct, a "write a function to check balanced brackets" request produced clean, idiomatic code with the right answers, and it explained the trade-offs of its own compression method accurately. Nothing garbled, no repetition, no collapse. The compression is impressive precisely because it is so hard to notice.

## The catch

Last month's post ended with two commands and [Ollama](https://ollama.com). This one cannot, and it is only fair to say so. The ternary format is custom — standard tools like Ollama and the usual `llama.cpp` build refuse to load it. You have to build the model author's own version of `llama.cpp` first:

```sh
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j
```

That compiles in a few minutes on a normal machine. Then download the 5.95GB model file and run it:

```sh
./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 65536 \
  -st -p "Summarise the document I am about to paste."
```

To open it up to the full 262K context, add a 4-bit cache: `-ctk q4_0 -ctv q4_0 -c 262144`.

It is a real trade-off: you buy the smaller size, the extra speed, and the huge context with a bit more setup, and a dependency on one team keeping their build going. For a quick try, last month's Ollama route is still the easier door. For a model you plan to lean on — long documents, private data, all day — this one earns the extra step.

## Where this leaves a small business

Five posts, one unchanged computer. The same RTX 3060 has gone from a single still image, to video, to video with sound, to a reasoning model that spilled over the edge of the card — and now to a 27B model that fits entirely on it, runs at a comfortable reading pace, and can swallow a novel's worth of your own documents in a single prompt, without a byte leaving the building.

That last part is the point. A context this large on a private, paid-for-once card means you can hand the model a whole contract, a full support-ticket history, or a year of notes and ask real questions about it — with none of it going to someone else's server, and no per-page cloud bill. The card did not get bigger. We just got much better at using it.

*It's a five-year-old graphics card. What it can do keeps surprising me.*


100a41bc708fc536b7140f337f846e8d7f5c6f80db39818675d03b244d8cbec2

How ternary quantization shrinks a 27B model to 5.95GB so it runs fully on a consumer RTX 3060, roughly four times faster than the 4-bit version, plus a measured look at exactly how much context fits in 12GB of VRAM: 64K by default, 128K or the full 262K with a quantized KV cache.

The fifth in our RTX 3060 series. The same Qwen3.8-27B we ran last month, re-quantized to under 6GB, now runs entirely on the 12GB card, about four times faster, and holds up to 262,000 tokens of context. A whole novel, on a five-year-old gaming card.

Now It Remembers: A 27B Model With a 262K-Token Context on the Same RTX 3060

{/* banner prompt: A playful, polished 3D clay-render in matte sage green, cream and terracotta: a stylized NVIDIA RTX 3060 graphics card holding a large traditional Chinese ink brush, painting a tiny misty mountain landscape onto a small scroll of cream rice paper, a little red seal-stamp prop and an ink stone beside it, soft studio shadows, simple pale background, 16:9. */}

Four posts into this series, the same RTX 3060 has learned to make
[images](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/),
[video](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/),
[video with sound](/blog/2026/08/04-now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/),
and even [reason like a 27B language model](/blog/2026/08/18-now-it-thinks-a-27b-language-model-on-the-same-rtx-3060/).
This time I wanted to point it at something harder to fake: a *specific,
demanding art tradition*. Not "an image in the style of," but **traditional
Chinese ink-wash painting — 水墨 (shui-mo) — on rice paper.**

It's a good test because the style is unforgiving. Real ink painting lives in
what *isn't* painted: the empty rice paper, the 留白 (liú bái) negative space,
the way a single dry-brush stroke suggests a whole pine tree. Get it slightly
wrong and it looks like a greeting card. So: can a five-year-old graphics card,
sitting in a home office, paint a convincing misty-mountain landscape? And how
close does it get to the cloud giants?

## The machine (still the same one)

Same RTX 3060, 12GB of VRAM, 64GB of system RAM, that this whole series is built
around — a card that shipped back in **2021**. Nothing upgraded.

## The test

One prompt, given to every model — a classic 山水 (shan-shui, "mountain-water")
scene:

> *Traditional Chinese ink wash painting on xuan rice paper: misty mountains, a
> lone fisherman's boat, gnarled pines on the cliffs, drifting fog, generous
> negative space, monochrome ink with grey gradients, a small red seal.*

Three models ran **locally on the 3060**. Three ran in the **cloud**, for
comparison.

## The local heroes — painted on the 3060

Here's the headline: **it works, and it works beautifully.** The trick isn't a
giant model — it's a tiny add-on. You take a general open image model already
running on the card and bolt on a **LoRA**: a small (~40–55MB) "style adapter"
trained on Chinese ink paintings. Drop the file in, add a trigger word, and the
model paints in that style.

**SDXL + a Chinese-ink LoRA** gave the most *authentic* result — loose,
expressive, with the faint colour washes and worn look of a scroll that has hung
on a wall for a century:

![A Chinese ink-wash landscape generated locally on an RTX 3060 with SDXL and a Chinese-ink LoRA: misty mountains, a small boat with two figures, pine trees, on aged rice paper](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/now-it-paints-chinese-ink-landscapes-on-the-same-rtx-3060/sdxl-ink.png)

**FLUX.1-dev + an ink-style LoRA** was the more *refined* look — crisper pines,
deeper atmospheric mist, a flawless composition. It reads as premium modern
"digital ink" rather than a genuine antique, but as an image it's stunning:

![A Chinese ink-wash landscape generated locally on an RTX 3060 with FLUX and the zyd232 ink-style LoRA: sharp misty peaks, pine trees on cliffs, a lone boatman on a still river](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/now-it-paints-chinese-ink-landscapes-on-the-same-rtx-3060/flux-zyd232.png)

Swap in a *different* ink LoRA and Flux shifts mood — softer, mistier, the peaks
dissolving further into fog — proof that the "look" is a small file you can
change, not a whole new model to download:

![A softer, mistier Chinese ink landscape from FLUX with a second ink-style LoRA (liujiyou), peaks dissolving into fog](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/now-it-paints-chinese-ink-landscapes-on-the-same-rtx-3060/flux-liujiyou.png)

Each took **about a minute** on the 3060. Both came out of a card that cost less
than a phone, in a home office, for the price of the electricity.

## The honest numbers

<Table data={
    {
        headers: ['', 'Local (RTX 3060)', 'Cloud'],
        rows: [
            ['Setup', 'Forge + a ~50MB LoRA file', 'An API key'],
            ['Time per image', 'About 1 minute', 'Seconds to a minute'],
            ['Cost', 'Electricity', 'A fee per image'],
            ['Privacy', 'Nothing leaves the room', 'Sent to a server'],
            ['Chinese calligraphy', 'Decorative gibberish', 'Legible (gpt-image)'],
            ['Best for', 'Authentic painterly landscapes', 'When you need real text and seals'],
        ]
    }
} />

## The cloud ceiling

To see how far ahead the big players are, I gave the same prompt to three cloud
models. This is where you see both the ceiling — and the one thing local can't
yet touch.

**OpenAI's gpt-image** was the standout of the entire test. Not just
museum-grade brushwork, but **real, legible calligraphy** — it wrote an actual
classical couplet by the poet Du Fu, with a proper signature line and seal
stamps. Look at the characters in the top-left; they're *real writing*, not
decoration:

![A highly authentic Chinese ink landscape by OpenAI gpt-image, with real legible calligraphy, a signature line and red seals](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/now-it-paints-chinese-ink-landscapes-on-the-same-rtx-3060/openai-gptimage.png)

**Zhipu's GLM (CogView-4)**, a Chinese model, produced a clean, textbook
shan-shui — confident and correct, if a little safe:

![A clean monochrome Chinese ink landscape by Zhipu GLM CogView-4](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/now-it-paints-chinese-ink-landscapes-on-the-same-rtx-3060/glm-cogview.png)

**Google's Gemini** did something unexpected — it rendered the painting as a
fully *mounted hanging scroll on a wall*, silk border and all, plus two zoomed
detail crops. Impressive, but more a product photo than a painting:

![Google Gemini's take, showing the ink painting as a mounted hanging scroll on a wall with detail crops](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/09/now-it-paints-chinese-ink-landscapes-on-the-same-rtx-3060/gemini.png)

## What I learned: the writing is the tell

Here's the interesting part. On the *painting* — the mountains, the mist, the
pines, the mood — the local 3060 results genuinely hold their own against the
cloud. Side by side, you'd struggle to pick the home-GPU images out of a lineup.

The gap is the **calligraphy.** Every local model painted beautiful *decorative*
brushstrokes where the inscription should be — squiggles that look like writing
but say nothing. Only gpt-image produced characters you can actually read. If
your painting needs a real poem or a legible seal, that's still a cloud job.
For everything else — the art itself — the card on your desk is enough.

## Where this leaves a small business

If you sell tea, run a restaurant, teach calligraphy, or just want a beautiful,
culturally-grounded header image for a newsletter, you no longer need a stock-art
subscription or a design agency. A modest computer you may already own can paint
you an original ink landscape in about a minute, privately, for the cost of the
power it draws — and you can generate a hundred variations until one is right.

The cloud still wins when the words on the page have to be real. But the *art*?
That quietly moved into the home office, onto a graphics card from 2021.

*Same card. Five years old. Now it paints.*


215c8cbaee653c59092628118433dd6fab0816de09b333db4b8e16dd3d1366b8

Generating traditional Chinese ink-wash painting (shui-mo / guohua) locally on a consumer RTX 3060 with SDXL and FLUX plus ink-style LoRAs, compared against cloud models (OpenAI gpt-image, Google Gemini, Zhipu GLM/CogView): real results, honest limits, and the one thing local models still can't do.

The fifth in our RTX 3060 series. We test whether a five-year-old 12GB graphics card can paint traditional Chinese ink-wash landscapes (shui-mo) on rice paper — locally, with open models — and pit the results against the cloud heavyweights.

Now It Paints: Chinese Ink Landscapes on the Same RTX 3060

{/* banner prompt: A playful, polished 3D clay-render in matte sage green, cream and terracotta: a stylized NVIDIA RTX 3060 graphics card with a softly glowing rounded brain resting on top of it, a few small thought-bubble and gear props floating beside it to suggest reasoning, soft studio shadows, simple background, 16:9. */}

The first three posts in this series were about *making* things — the same RTX 3060
learning to generate [images](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/),
then [video](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/),
then [video with sound](/blog/2026/08/04-now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/).
This one is different. This time the card isn't drawing or filming anything — it's
*thinking*. I put a **27-billion-parameter language model on the same 12GB card**,
and it works.

The headline: **Qwen3.8-27B**, a reasoning model released only a few days ago, runs
locally on a consumer RTX 3060 that shipped back in **2021**. It answers questions, writes code,
reasons step by step — and it can even look at a picture and tell you what's in it.
The kind of model people assume needs a data centre, or at least a $2,000 graphics
card, runs on hardware a small business already owns.

## The machine (still the same one)

Same box as every post in this series: an RTX 3060 with **12GB of VRAM**, 64GB of
system RAM, Debian. Nothing upgraded. The card is the constant.

## Why this is surprising

A 27B model is *big*. The rule of thumb people quote is that you need a 24GB card
(an RTX 4090 or better) to run one at all — and its full-precision form is over
50GB, firmly data-centre territory. So "27B on 12GB" sounds like it shouldn't be
possible.

It is, because of two things working together: **quantization** and **offloading.**

## Fitting an 18GB model into 12GB

Quantization compresses the model's numbers to 4 bits each, shrinking it from 50GB+
down to about **18GB** with very little quality lost. That still doesn't fit in
12GB — but it doesn't have to. The card holds a bit more than half the model
(**about 57%**), and the rest lives in the 64GB of system RAM, streamed onto the
GPU as it's needed. The GPU does the heavy math; the spare layers wait in RAM.

The remarkable part is how *easy* it was. The video-with-sound model in the last
post took patches and careful memory tuning. This time it was genuinely two
commands — download, and run. The tool (Ollama) works out the GPU/RAM split
automatically.

## The honest numbers

<Table data={
    {
        headers: ['', 'On the RTX 3060'],
        rows: [
            ['Model', 'Qwen3.8-27B (released this month)'],
            ['Size on disk', '~18GB (4-bit)'],
            ['Runs on GPU', '~57% on the card, ~43% streamed from RAM'],
            ['VRAM used', '11 of 12 GB'],
            ['Speed', '~4.7 words-per-second (reads about as fast as you do)'],
            ['Can it see images?', 'Yes — it is multimodal'],
            ['Best for', 'Questions, coding help, explanations, one-image analysis'],
        ]
    }
} />

Four-ish words a second isn't fast — you watch the answer type itself out, roughly
at reading speed. This is not the tool for churning through a thousand requests an
hour. It's the tool for asking one good question and getting a thoughtful answer,
privately, on your own machine, for the cost of the electricity.

## It reasons — out loud

Qwen3.8 is a *reasoning* model: before it answers, it thinks. Ask it a coding
question and you can watch it work the problem — considering approaches, catching
its own mistakes, checking its logic — then give a clean final answer. Ask it to
find the longest palindrome in a string, and it doesn't just spit out code; it
reasons about edge cases and complexity first. That visible thinking is a big part
of why these newer models are so much more useful than the ones from a year ago.

## And it sees

Here's the part that closes the loop on this whole series. The model is multimodal,
so I showed it one of the AI-generated animal photos from our image experiments and
asked: *is this a real photograph, or AI-generated?*

It nailed it. It described the scene correctly — a brown bear in a rushing river,
autumn forest blurred behind it — and then, unprompted, listed exactly *why* it
looked artificial: fur that was "suspiciously smooth and painted," the
"characteristic milky AI bokeh" in the background, "melted" rocks, a glassy eye
lacking real reflections. A local model, on a 12GB card, giving a genuinely sharp
critique of a locally-generated image. The tools in this series can now check each
other's work.

## Try it yourself

If you have an RTX 3060 (or any 12GB card) and [Ollama](https://ollama.com)
installed, this is the whole thing:

```bash
ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M
```

The first run downloads the model; after that it's instant to start. Ask it
anything. To feed it an image, just put a file path in your message.

## Where this leaves a small business

Four posts, one unchanged computer. The same RTX 3060 went from a single 512px
still, to short video, to video with a soundtrack — and now to a reasoning model
that can answer questions, help write code, and read an image, all without a single
byte leaving the building.

That last point matters more than the speed. Running the model *locally* means your
questions, your customer data, your documents never touch someone else's server.
No per-message cloud bill, no data leaving your office, no dependency on a
subscription that can change its terms next quarter. For a lot of small businesses,
"good enough, private, and paid-for-once" beats "fastest" every time.

It isn't a data centre. It doesn't need to be.

*It's a five-year-old graphics card. What it can do keeps surprising me.*


948ef2e0820c8a951e9aa975e9f905c44c461234611a88494f1617dd4622ea0a

Running Qwen3.8-27B, a brand-new 27B multimodal reasoning model, locally on a consumer RTX 3060 with only 12GB of VRAM: how an 18GB model fits, honest tokens-per-second, what it can and can't do, and the one command to try it yourself with Ollama.

The fourth in our RTX 3060 series. Qwen3.8-27B — a 27-billion-parameter reasoning model released this month — runs on the same 12GB card. Not in the cloud, not on a $2,000 GPU. One command, and it thinks, codes, and even sees.

Now It Thinks: A 27B Language Model on the Same RTX 3060

{/* banner prompt: A playful, polished 3D clay-render in matte sage green, cream and terracotta: a stylized NVIDIA RTX 3060 graphics card as a tiny cinema, a small strip of film curling out of it, and a rounded speaker or sound-wave motif beside it emitting soft concentric rings to show the video now has audio; a little clapperboard prop, soft studio shadows, simple background, 16:9. */}

I ended the [last post](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/)
with a promise: the same RTX 3060 that learned to make short video would next try
to make video *with sound* — using **MiniMax H3**, an open-weights model that
generates its own audio. The weights landed publicly, I put them on the same
card, and this is what actually happened.

The headline first: it works. A single product photo goes in, and a few-second
clip comes out — the model turning, the fabric settling, a soft spoken line and
room tone — **audio the model generated itself**, not dubbed on afterwards. On a
12GB consumer card. That part genuinely surprised me.

## The machine (still the same one)

Same box as the [image retrospective](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/)
and the [video post](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/):
an RTX 3060 with 12GB of VRAM, 64GB of system RAM, Debian. Nothing upgraded. The
card is the constant this whole series is built around.

## What "video with sound" means here

Most open video models are mute — you get moving pictures and add sound in an
editor. MiniMax H3 is different: video and audio come out of the *same* model,
generated together, so the soundtrack matches the scene. Describe waves and a
spoken line, and you get waves and a voice. That single capability is why it was
worth the effort.

## Fitting 54GB into 12GB

Here is the honest part. H3 is big: about **54GB of weights** — a 21GB video
model, a 27GB text encoder, and two decoders for picture and sound. That does
not fit in 12GB of VRAM. Not close.

It runs anyway because the weights live in the 64GB of system RAM and stream onto
the card a slice at a time, keeping just enough on the GPU to do the next step.
It is slow, but it finishes. Two smaller things also got in the way: the model is
new enough that its picture and sound decoders each had a small bug that had to
be patched before a clip would complete, and the card needed a healthy slice of
memory kept free so the compressed weights had room to unpack. Fiddly, but
one-time — once it was set up, it just ran.

## The result: a product clip that turns and talks

This is the workflow a small business would actually use: start from one product
photo, and let the model animate it — the person turns to camera, the knit
settles, a quiet line of audio plays — while the product stays readable. Here is
a real clip generated on this card. **Press play with the sound on** — the audio
came out of the model, not an editor.

<video controls playsInline preload="metadata" poster="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/sweater-poster.png" style={{ display: "block", width: "100%", maxWidth: "340px", margin: "1.5rem auto", borderRadius: "10px" }}>
  <source src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/sweater-clip.mp4" type="video/mp4" />
</video>

And the same clip laid out frame by frame — a natural turn from side-on to
camera, the knit texture and anatomy holding through the whole move:

![Frames from a MiniMax H3 clip: a model in a cream knit sweater turning to camera, generated with sound on an RTX 3060](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-talks-open-source-video-with-sound-on-the-same-rtx-3060/filmstrip-sweater.png)

## The honest numbers

<Table data={
    {
        headers: ['', 'On the RTX 3060'],
        rows: [
            ['Clip length', 'A few seconds (2-5s)'],
            ['Resolution', '512-640px'],
            ['Audio', 'Yes - native stereo, generated with the video'],
            ['Model size', '~54GB of weights, streamed from system RAM'],
            ['Time per clip', 'Roughly 20-27 minutes'],
            ['Best for', 'A short product clip with a voice or ambient soundtrack'],
        ]
    }
} />

Twenty-odd minutes for a few seconds is the real cost of streaming that much
weight through a small card. This is not something you batch out by the hundred
on a 3060 — it is something you run overnight for one good clip. But the clip has
sound, and it cost electricity.

## What I learned about the "AI look"

The most useful lesson wasn't about speed — it was about *why* AI video still
reads as AI, and how to fight it.

Image-to-video can never look better than the photo you start from; it inherits
that frame's every quirk and then adds motion on top. My first attempts started
from bright, high-contrast outdoor stills — and they had that unmistakable
"AI feel": waxy skin, hard outlines, the joins between body parts subtly wrong.
The motion clung to those hard edges and made them worse.

The clip above started from a **soft, studio-lit photo** instead — diffused
light, a plain background, gentle edges. And it held together: real fabric
texture, natural movement, believable anatomy through the whole turn. Same model,
same card, completely different result.

So the practical rule, if you try this yourself: **start from soft, evenly-lit
photos, not harsh high-contrast ones.** The model can't cling to hard edges that
were never there. It's the single biggest lever on how "real" the output looks —
bigger than any setting.

## Where this leaves a small business

Three posts, one unchanged computer. The same RTX 3060 went from struggling with
a single 512px [still](https://www.tyolab.com/prompts), to short
[silent video](/blog/2026/08/03-now-it-moves-video-generation-on-the-same-rtx-3060/),
to a short clip **with its own soundtrack** — a product turning to camera and
speaking a line, generated from one photo, locally, for the cost of the power it
drew.

It is not one-click, and it is not fast. But the gap between "I have a product
photo" and "I have a short product video with sound" has quietly closed on
hardware a small business can already afford. That's the whole point of this
series: the tools that used to need a studio now need a modest computer and a bit
of patience.

*The card hasn't changed in two years. What it can do has.*


0e5f273a081f550b8cae1cc64cfacb00dc1a76e174947517e86cf2fa03d0d6a2

Running MiniMax H3, an open-source video model with native audio, on a consumer RTX 3060 12GB: how 54GB of weights fit in 12GB, honest generation times, the quality limits, and turning one product photo into a clip with sound.

The third in our RTX 3060 series. MiniMax H3, an open-weights video model that generates its own audio, runs on the same 12GB card: a product clip with a real soundtrack, from a single photo, generated locally.

Now It Talks: Open-Source Video With Sound on the Same RTX 3060

{/* banner prompt: A playful but polished 3D clay-render, matte materials in sage green, cream and terracotta, of a stylized NVIDIA RTX 3060 graphics card acting as a small film projector on a simple background, casting a beam of light onto a tiny cinema screen that shows a strip of film frames; a couple of small clapperboard and film-reel props beside it, soft studio shadows, 16:9. */}

In the [last post](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/)
I traced how open-source *image* generation went from a 512px struggle to
photoreal Flux on one RTX 3060. I ended it with a tease about video. Here's the
payoff — and it's the part that genuinely surprised me.

Two years ago, generating video on a consumer graphics card wasn't really a
thing; the early attempts were a few flickery, warping frames. The **same RTX
3060** now produces short, coherent clips — and, more usefully for a small
business, it can take a **single product photo and turn it into a promo**.

## The machine (still the same one)

Same box as the [image post](/blog/2026/08/03-two-years-of-open-source-image-ai-on-one-rtx-3060/)
and the [LLM benchmark](/blog/2026/05/11-64gb-ram-12gb-vram-the-honest-local-llm-benchmark/):
an RTX 3060 with 12GB of VRAM, 64GB of system RAM, Debian 13. Nothing upgraded.

## What runs today

Two open models do the heavy lifting, both locally:

- **Wan 2.2** — higher quality, does text-to-video *and* image-to-video. Slower.
- **LTX-Video** — faster and lighter, good for quick drafts.

The 12GB card can't touch the giant cloud video models, but with the weights
quantised and the text encoder offloaded to system RAM, these run — a few
seconds of 480–832px video per clip.

## The trick that matters: one photo becomes a promo

This is the workflow a small business will actually use. Generate a product
photo (or use one you already have), then animate *it* — the model, the fabric,
the light all start moving, while the product stays put. Each strip below is
frames from a real clip, and each one started as a single still generated on
this same card.

A knit sweater: the model turns to camera, hair catching a breeze.

![Frames from the sweater promo clip](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-moves-video-generation-on-the-same-rtx-3060/filmstrip-sweater.png)

Sunscreen — very on-brand for an Australian summer — the product held steady in
focus while the scene breathes:

![Frames from the sunscreen promo clip](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-moves-video-generation-on-the-same-rtx-3060/filmstrip-sunscreen.png)

And eyewear, a slow confident turn in golden-hour light:

![Frames from the sunglasses promo clip](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/08/now-it-moves-video-generation-on-the-same-rtx-3060/filmstrip-shades.png)

## The honest numbers

<Table data={
    {
        headers: ['', 'On the RTX 3060'],
        rows: [
            ['Clip length', 'A few seconds (2-5s)'],
            ['Resolution', '480p to ~832px'],
            ['Time per clip (Wan 2.2)', 'Roughly 4-8 minutes'],
            ['Time per clip (LTX)', 'Under a minute for a short draft'],
            ['Best for', 'Image-to-video product promos, social clips'],
            ['Audio', 'Not yet — these models are silent'],
        ]
    }
} />

It's not one-click cinema. Big, fast motion still warps; the sweet spot is
*subtle* movement — a turn of the head, a breeze, a slow push — with the product
kept steady. You generate a couple of takes and keep the best, same as with
stills. But a usable, on-brand product clip in a few minutes, for the cost of
the electricity, on a card that couldn't manage one clean still two years ago —
that's the shift.

## What's next: sound

The current open video models are mute. The next one on our bench, **MiniMax
H3**, ships open weights *with* native audio and is reported to run on exactly
this class of hardware. When its weights land publicly, we'll put it on the same
3060 and see whether a small business can generate a product clip **with a
soundtrack** end to end — and we'll write up what actually happens.

For now, the takeaway from two posts and one unchanged GPU: the same modest
computer that makes studio-grade [stills](https://www.tyolab.com/prompts) now
makes short video too, and the gap between "I have a product photo" and "I have a
product video" has quietly closed.

*Next up: MiniMax H3 — open-weights video with sound — on the very same card.*


492ad0a4331b0c870b48208cef2d272462372bd58c519f5bd6ce2c257a5eeccb

How open-source video generation runs on a consumer RTX 3060 12GB in 2026: Wan 2.2 and LTX-Video for text-to-video and image-to-video, real generation times, VRAM limits, and a worked example turning a product photo into a promo clip.

The follow-up to our image retrospective: the same RTX 3060 that struggled with a single 512px still two years ago now generates short video — including product promos animated from a single photo — with open models like Wan 2.2 and LTX-Video.

Now It Moves: Video Generation on the Same RTX 3060

{/* banner prompt: A playful but polished 3D render of a stylized RTX 3060 as a small factory on a simple background, using matte clay-like materials in a color story of sage green, cream, and terracotta; a conveyor belt on the left carries misshapen, glitchy figures, while a belt on the right carries perfectly formed photorealistic objects. */}

Eighteen months ago I wrote a tutorial on
[mastering human portraits with Stable Diffusion](/blog/2025/02-06-mastering-human-portraits-with-stable-diffusion-ai/).
Reading it back is a little painful. It opens with "trust me, this is hard," and
its list of search keywords reads like a bug report: *two heads, extra heads,
deformed face, extra eyes, disfigured, duplicate.* Those weren't edge cases —
they were what you fought on every render.

This week I ran open-source image generation on the **exact same computer**. Same
RTX 3060, same 12GB of VRAM. The only thing that changed is the software, and the
gap is hard to overstate.

## The machine (deliberately unchanged)

It's the same box from our [local-LLM
benchmark](/blog/2026/05/11-64gb-ram-12gb-vram-the-honest-local-llm-benchmark/):

- **CPU:** Intel Core i7-12700F
- **RAM:** 64GB DDR4
- **GPU:** NVIDIA GeForce RTX 3060 — 12GB VRAM
- **OS:** Debian 13

Holding the hardware constant is the whole point. Any improvement you see is the
open-source ecosystem getting better, not a new graphics card.

## Early 2025: Stable Diffusion 1.5, 512 pixels, and a war on anatomy

Back then, the state of the art you could actually run at home was Stable
Diffusion 1.5. My old parameters tell the story: 512×512, Euler a, CFG 7. To get
anything lifelike you leaned on community finetunes (RealisticVision, ChilloutMix,
F222), stacked LoRAs, and wrote paragraph-long negative prompts to *suppress* the
model's habit of adding a second head or a sixth finger. Then you generated a
dozen and kept the one that wasn't broken. It worked, but it was a craft — and a
grind.

## Mid 2026: the same card, a different league

Here's a portrait from the same 3060 today, straight out of the model —
no inpainting, no ControlNet, no clean-up:

![A photoreal portrait generated on the RTX 3060 in 2026](https://www.tyolab.com/api/tynion/images/f9f8fb60-d635-1048-cb10-c3492d7a4c4a/file)

Three things changed the game. **Flux** and modern **SDXL** finetunes replaced
SD 1.5 as the open baseline. They render natively at 1024px and up, and they get
hands, faces and proportions right the first time far more often than not.

They also learned to spell — the single clearest "we crossed a line" moment.
Readable text in an image was essentially impossible in 2025:

![A shopfront with legible hand-painted signage, generated locally](https://www.tyolab.com/api/tynion/images/3fa3d0ac-f3af-f6e7-952e-27b04f9b545e/file)

And the range on one card is wide — photoreal wildlife and clean anime styles
from the same machine, same afternoon:

![A photorealistic wildlife shot from the same GPU](https://www.tyolab.com/api/tynion/images/adbceadb-bc09-46f0-c8c7-bd76f312361f/file)

![A detailed anime illustration from the same GPU](https://www.tyolab.com/api/tynion/images/3d737b80-9cbb-186f-ce2c-c496fb918589/file)

## Then and now, side by side

<Table data={
    {
        headers: ['', 'Early 2025', 'Mid 2026'],
        rows: [
            ['Best model you can run', 'Stable Diffusion 1.5 + finetunes', 'Flux, SDXL finetunes'],
            ['Native resolution', '512 x 512', '1024px and up'],
            ['Anatomy', 'Fighting extra heads, hands, eyes', 'Right on the first try, mostly'],
            ['Text in the image', 'Gibberish', 'Often readable'],
            ['Keepers per ten tries', 'One or two', 'Most of them'],
            ['Effort per shot', 'Long negative prompts, LoRAs, inpainting', 'A plain-English sentence'],
            ['Video', 'Not on this card', 'A few seconds, locally'],
        ]
    }
} />

## The part that was pure science fiction in 2025: video

The same RTX 3060 now generates **video**. Wan 2.2 and LTX-Video produce short
text-to-video and image-to-video clips locally — a five-second scene in a few
minutes, on a card that two years ago struggled with a single 512px still. The
next open model on our list, MiniMax H3, even adds a synchronised soundtrack.

## What actually changed — and why it matters

It wasn't the GPU. Three shifts in the open-source world did the work:

- **Better base models.** Flux and SDXL are simply generations ahead of SD 1.5.
- **Quantisation.** Techniques like fp8 and 4-bit weights shrink 12-billion-parameter
  models enough to fit inside 12GB of VRAM with quality mostly intact.
- **Smarter memory management.** The tools now offload the heavy text encoders to
  system RAM, so the GPU only holds what it must.

Put together, a consumer card that launched years ago runs models that didn't
exist when you bought it — because the community learned to make them fit.

For a small business or a hobbyist, that's the real headline: the *same* modest
hardware budget now buys studio-grade stills and short video, from software
that's free and runs on your own desk. Every image in this post was generated on
that one 3060; you can browse and remix the prompts in our
[gallery](https://www.tyolab.com/prompts).

*Next up: putting MiniMax H3 — open-weights video with native audio — on the very
same card, and seeing how far 12GB can really go.*


3b1c2bde5a7d34a1dd6cf7101d23b93c2774ca890ffc4eec11944f83608ebd17

A hands-on before-and-after: how open-source text-to-image models improved on the exact same RTX 3060 12GB between early 2025 and 2026 — Stable Diffusion 1.5 vs Flux and SDXL finetunes, resolution, anatomy, text rendering, hit-rate, and video.

We ran open-source image generation on the same RTX 3060 we used 18 months ago. The hardware is identical; the results are unrecognisable — from 512px portraits fighting extra heads to 1024px Flux, near-perfect anatomy, readable text, and even video.

Two Years of Open-Source Image AI on One RTX 3060


{/* banner prompt: A detailed architectural blueprint diagram of a system for local LLM inference, showing a stylized RTX 3060 GPU with its 12GB VRAM as a partitioned grid, connected by flow-lines to various model blocks (GGUF), with technical annotations and performance metrics in a clean, structured layout on a cyanotype blue background.

Additional instructions: with some construction workers watching */}

I've been building a local AI inference server at home. Not for any grand reason — mostly curiosity, a bit of privacy paranoia about sending everything to OpenAI, and the guilty pleasure of watching a model load onto my own GPU. After a few weeks of downloading models, writing benchmark scripts, and debugging scoring bugs at midnight, I have real numbers to share.

This is not a theoretical comparison. Every number here came from actual inference on my machine.

## The Hardware

- **CPU:** Intel Core i7-12700F (12 cores, 20 threads)
- **RAM:** 64GB DDR4
- **GPU:** NVIDIA GeForce RTX 3060 — 12GB VRAM
- **OS:** Debian 13
- **Inference engine:** [llama.cpp](https://github.com/ggerganov/llama.cpp) in router (multi-model) mode

The RTX 3060 with 12GB VRAM is the constraint that shapes everything. It's a solid consumer card — fast, quiet, widely available — but 12GB means you need to think carefully about which models you can load, at what quantization, and at what context length.

## The Setup: llama.cpp Router Mode

Rather than running a separate server process for each model, I use llama.cpp's `--models-dir` router mode. Point it at a directory of GGUF files, set `--models-max 2` (keep up to 2 models hot in VRAM at once), and it handles loading and unloading automatically per request. It exposes an OpenAI-compatible API at `http://localhost:8080/v1`.

The systemd service looks like this:

```ini
[Service]
ExecStart=/opt/llama.cpp/build/bin/llama-server \
    --host 0.0.0.0 --port 8080 \
    --models-dir /data/models \
    --models-max 2 \
    --threads 16 \
    --ctx-size 8192 \
    --n-gpu-layers 99 \
    --cont-batching \
    --flash-attn auto \
    --log-disable
```

`--n-gpu-layers 99` pushes every layer it can onto the GPU. `--flash-attn auto` enables Flash Attention when the model supports it. Models that don't fit entirely in VRAM will spill some layers to CPU — that's slow, and it shows up in the benchmark numbers.

## The Benchmark

I wrote a small Python script that tests each model on three task categories:

- **GSM8K** — 10 grade-school math word problems. Tests arithmetic reasoning.
- **MMLU** — 10 multiple-choice questions across medicine, history, philosophy, science, and CS. Tests general knowledge.
- **HumanEval** — 5 Python coding tasks (sum_to_n, palindrome check, fizzbuzz, count vowels, flatten). Tests code generation.

Each request uses `temperature=0` for determinism. Token speed (`t/s`) is measured from the `timings.predicted_per_second` field in the llama.cpp response.

### Handling Thinking Models

Several models — Qwen3, DeepSeek-R1-Distill, Qwen3.5 — are *thinking* or *reasoning* models that generate an internal chain of thought before giving a final answer. This created two real scoring headaches:

1. **llama.cpp routes Qwen3's thinking to a separate `reasoning_content` field** in the API response, leaving `content` empty. A naive scorer reading only `content` scores every Qwen3 response as 0%.

2. **256 max_tokens is nowhere near enough.** Thinking models use most of their budget on reasoning and never output a final answer. I bumped to 1024 tokens for GSM8K/MMLU and 2048 for coding tasks.

3. **MMLU first-match failure.** A reasoning model might mention option "A" in its thinking ("let's rule out A...") before concluding "the answer is B". Using `re.search` to find the first letter match gets the wrong answer. I switched to `re.findall` and take the *last* `[ABCD]` match.

The fixed scoring functions:

```python
def chat(model, prompt, max_tokens=512):
    r = requests.post(BASE_URL, json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "temperature": 0,
    }, timeout=120)
    data = r.json()
    msg = data["choices"][0]["message"]
    # Qwen3 thinking models: answer is in reasoning_content, content is empty
    text = msg.get("content") or msg.get("reasoning_content", "")
    tps = data.get("timings", {}).get("predicted_per_second", 0)
    return text, tps

def score_mmlu(response, answer):
    stripped = strip_thinking(response)   # remove <think>...</think>
    for text in [stripped, response]:     # fallback to full text if strip leaves nothing
        matches = re.findall(r'\b([ABCD])\b', text.strip().upper())
        if matches and matches[-1] == answer:
            return True
    return False
```

Getting these details right took most of the iteration time. The first run showed Qwen3-8B at 0% on everything — not because it's a bad model, but because the scoring was broken.

## The Results

All models run as Q4_K_M quantization unless noted. 14B models were tested at `--ctx-size 4096` (reduced from 8192) to fit the KV cache alongside the ~8.4GB weights within 12GB VRAM.

```
Model                                    Params  GSM8K   MMLU   Code    Avg    t/s
─────────────────────────────────────────────────────────────────────────────────
Meta-Llama-3.1-8B-Instruct-Q4_K_M        8B     80%    80%   100%    87%   64.5
Qwen2.5-Coder-14B-Instruct-Q4_K_M       14B     80%    80%   100%    87%   35.8
Llama-3.2-3B-Instruct-Q4_K_M             3B     70%    80%   100%    83%  128.3
Qwen2.5-Coder-7B-Instruct-Q4_K_M         7B     40%   100%   100%    80%   69.3
Qwen_Qwen3-14B-Q4_K_M                   14B     50%    70%   100%    73%   33.4
Llama-3.2-1B-Instruct-Q8_0               1B     50%    50%   100%    67%  211.9
Qwen_Qwen3-8B-Q4_K_M                     8B     30%    70%   100%    67%   59.5
DeepSeek-R1-Distill-Qwen-7B-Q4_K_M       7B     80%    50%    20%    50%   66.2
Qwen3.5-9B-Q4_K_M                        9B     20%    70%    20%    37%   49.9
Qwen3.5-4B-Q4_K_M                        4B     30%    70%     0%    33%   77.1
MiMo-VL-7B-RL-q4_k_m                    7B     10%    50%    20%    27%   65.6
MiMo-7B-RL-Q4_K_M                        7B     10%    60%     0%    23%   64.2
THUDM_GLM-Z1-9B-0414-Q4_K_M              9B      0%     0%     0%     0%   51.3 †
─────────────────────────────────────────────────────────────────────────────────
DeepSeek-R1-Distill-Qwen-14B-Q4_K_M     14B    FAIL — wouldn't load at ctx=4096
openai_gpt-oss-20b-Q4_K_M               20B    FAIL — ~11.5GB, exceeds 12GB VRAM
Kimi-Linear-48B-A3B-BPW2.6              48B    FAIL — corrupt download
```

† GLM-Z1 uses non-standard thinking tags that our scorer doesn't strip. Its real capability is unknown from this run.

## What Surprised Me

### The 3B Llama Punches Hard

Llama-3.2-3B-Instruct at 128 t/s and 83% average is the sleeper hit. It's nearly as capable as the 8B model, runs at twice the speed, and fits comfortably in VRAM. If you're building something latency-sensitive — a coding assistant, a quick-reply bot — this is the model I'd reach for first.

### Qwen2.5-Coder-7B Is Surprisingly Well-Rounded

This model scored 100% on both MMLU *and* Code, and 40% on math. Named "Coder" but it's not just good at code — it's one of the most reliable general-purpose models in the lineup. At 69 t/s it's fast too.

### DeepSeek-R1-7B Is a Math Specialist, Not a Generalist

80% on GSM8K math — tied with the best models in the field. But 50% on MMLU and only 20% on coding. The reasoning focus pays off for math and pays a price everywhere else. Use it when you specifically need step-by-step numerical reasoning.

### 20B Models Don't Fit

The `openai_gpt-oss-20b` model is about 11.5GB at Q4_K_M quantization. That leaves only 500MB for the KV cache — not viable. It loaded partially with CPU offload, but responses were too slow and wrong. If you want a 20B+ model, you either need more VRAM or a heavily quantized version (Q2 or less), which carries its own quality tradeoffs.

### 14B Models Need Careful Context Sizing

At `--ctx-size 8192`, 14B Q4_K_M models fail to load — the weights (~8.4GB) plus the KV cache exceed 12GB. Dropping to `--ctx-size 4096` solved it for Qwen3-14B and Coder-14B. DeepSeek-R1-14B refused to cooperate even at 4096 — likely a memory fragmentation or architecture-specific issue.

### GLM-Z1 Needs a Different Scorer

GLM-Z1 almost certainly isn't as bad as our 0% score suggests. It uses thinking token formats that aren't `<think>...</think>` tags, so our strip function leaves garbage in the text. This is on the benchmark, not the model. Something to fix in a future run.

### Kimi-Linear-48B: So Close, Yet So Far

Kimi-Linear-48B is a Mixture-of-Experts model with 48B total parameters but only 3B active per token — meaning it should fit in 12GB VRAM and run at reasonable speed. At BPW2.6 quantization the file is about 8.1GB. We downloaded it twice. The first download (via `hf download`) produced a file with 6.9MB of zeros at the start — invalid GGUF magic. The second download via `wget` direct URL is in progress. Watch this space.

## The Hy-MT1.5 Translation Models

We also downloaded the full set of [Hy-MT1.5](https://huggingface.co/hy-mt) translation models from Tencent — 1.8B and 7B variants in multiple quantizations. These aren't benchmarked here because GSM8K/MMLU/HumanEval aren't the right tests for a translation model. A separate post will cover those with bilingual evaluation tasks.

## Practical Takeaways

If you're setting up a similar server, here's what I'd do differently from day one:

**1. Start with Llama-3.2-3B.** It's the best quality-per-token-per-second model in this lineup. Keep it hot in VRAM for low-latency tasks.

**2. Add Qwen2.5-Coder-7B for general use.** It's well-rounded across knowledge, code, and fast enough for interactive use.

**3. Use llama.cpp router mode from the start.** Running one server that hot-swaps models is far better than spinning up individual servers. You get an OpenAI-compatible API, automatic model loading, and a simple `/v1/models` endpoint to see what's available.

**4. Budget VRAM carefully for 14B models.** You need `--ctx-size 4096` (not 8192) to fit 14B Q4_K_M models. That's fine for most tasks, but be aware of the tradeoff.

**5. Validate GGUF downloads before benchmarking.** Check the first four bytes: they should be `47 47 55 46` (ASCII "GGUF"). A quick `od -A x -t x1z model.gguf | head -1` saves hours of debugging.

**6. Fix your thinking-model scorer before running anything.** The `reasoning_content` split in llama.cpp, the max_tokens floor, and the last-match MMLU scoring all need to be right or you'll think every thinking model is broken.

## What's Next

- Re-test GLM-Z1 with a proper thinking-tag stripper
- Kimi-Linear-48B, if the second download produces a valid file
- Translation quality tests for Hy-MT1.5
- A comparison post once larger VRAM cards (24GB+) are in reach

The benchmark script is straightforward Python using the OpenAI-compatible API — if you want to run it against your own setup, the core logic is about 100 lines. Happy to share it.

---

*Tested on Debian 13, llama.cpp built from source (May 2026), RTX 3060 12GB, i7-12700F, 64GB RAM.*


dc5102b82d71cb9e13d63287133417768b9af813bb6643140e41d8bee1da88f4

A hands-on benchmark of 13 local LLMs — Llama, Qwen3, DeepSeek-R1, Coder models and more — running on a home server with 64GB RAM and RTX 3060 12GB VRAM via llama.cpp. Includes GSM8K math, MMLU knowledge, and HumanEval coding scores, real token speeds, and an honest account of what doesn't fit.

We benchmarked 13 local LLMs on a machine with 64GB RAM and an RTX 3060 12GB using llama.cpp router mode — here's what scored well, what failed to load, and what surprised us.

Tag: Local Ai